Method for generating and analyzing an overview contrast image

ABSTRACT

A method for generating and analyzing an overview contrast image of a specimen carrier and/or of specimens situated on a specimen carrier. A specimen carrier arranged at least partially in the focus of a detection optical unit is illuminated in transmitted light using a two-dimensional, array-like illumination pattern. At least two overview raw images are detected using different illuminations of the specimen carrier, and, according to information to be extracted from the overview contrast image, a combination algorithm is selected by means of which the at least two overview raw images are combined to form the overview contrast image. According to information to be extracted from the overview contrast image, an image evaluation algorithm is selected by means of which the information is extracted.

FIELD OF THE INVENTION

The invention relates to a method for producing and analyzing anoverview contrast image of a sample carrier and/or of samples arrangedon a sample carrier.

BACKGROUND OF THE INVENTION

Traditional light microscopes have been developed on the assumption thata user can look through an eyepiece onto a sample carrier while sittingor standing and interact directly with the sample in the sense that hewill quickly be able to obtain an overview of the sample and of thefield of view of the objective and laterally move the sample carrierwith the sample either directly or using an adjustable sample stage soas to bring other regions of a sample into the field of view of theobjective. The user of the microscope can remain at his location andneed only minimally move his head, which means that in this respect,traditional microscopes are highly ergonomic.

Examination methods in particular for biological samples have beendeveloped further over time, and as a consequence, the construction ofmicroscopes that are suitable for performing said examination methodshave also become ever more complex. In today's microscope systems, whichpermit the recording of image stacks along the observation direction andthe reconstruction of a spatial image of the sample therefrom, imagesare produced with the aid of detectors. Detectors that are used are, forexample, cameras that are configured with corresponding surface sensors,or photomultipliers. Consequently, the working space in such systems hasshifted from the microscope stand and consequently from the sample tothe computer or screen of such a computer. And yet, the working space infront of the stand is likewise used and required to prepare, i.e. setup, the sample for the examination. To this end, the sample on thesample carrier must be moved into the field of view of the objective, asample region must be chosen, an adjustment to the position of saidsample region must be performed, and, finally, the sample must bebrought into focus. The workflow when using modern, complex microscopesystems is therefore associated with two working spaces that representdifferent steps in the workflow and are spatially separate from oneanother—first, the microscope stand with the eyepiece for directobservation, and then the screen of a connected computer.

By attaching further instruments such as incubators for examining livingcells, the direct view onto the sample, that is to say the position ofthe objective field of view in the sample, is greatly limited. Ifmoreover larger sample carriers—for example multiwell plates—are used sothat a multiplicity of samples can be examined in succession, theorientation on the sample is likewise obstructed.

As a result, finding the sample and setting the sample region are mademore difficult for a user, and in addition, there is an orientation losson the sample. Changing repeatedly between the computer working spaceand the microscope stand at which the sample can be directly observed isnecessary for setting purposes.

In addition, microscope systems that are geared at the throughput of ahigh number of samples and are not continuously controlled also exist.In this case, sample regions must be automatically detected andmicroscopically captured. The sample carriers here typically carry aninscription, for example in the form of a barcode or a QR code, thatmust be assigned to the samples. Such examinations proceedsemi-automatically; for example, a user only intervenes to change thesample carriers, to set an overview image, or to register the samplecarrier number.

Especially when using multiwell plates in microscope systems with a highthroughput, some of the wells may contain no samples or may containincorrectly embedded, contaminated or defective samples. Said wells arelikewise examined in the semi-automatic method, even though the resultscannot be used, which means that the process takes more time than isactually necessary.

Although different methods exist in the prior art for producing overviewimages, these have more or less significant drawbacks. For example, animage can be recorded with the microscope optical unit and a weaklymagnifying objective and a camera that is arranged downstream of theobjective. However, this allows the recording of merely a small objectfield as compared to the size of the sample or of the sample carrier, inparticular if sample carriers for a plurality of samples are used. Inorder to be able to record large sample regions, such as for example formultiwell sample carriers, it is therefore necessary to record aplurality of images of sections of the sample or of the sample carrierthat are located next to one another and to subsequently combine them.This procedure is quite lengthy and not suitable for example forexamining living samples.

An overview image can also be recorded if, instead of the microscopeoptical unit, a camera with a camera objective lens with which arelatively large object field can be imaged is used; the cameraobjective lens is generally not telecentric. This solution isimplemented for example in the AxioScan series by the applicant, but canbe used only with bright field illumination and incident light.Coverslips and undyed samples can be detected only with difficulty inthis method.

Different solutions illuminate samples and sample carriers obliquely,i.e. at an angle that differs from zero relative to the optical axis,wherein the back-scattered light is detected. The sensitivity of themeasurement over the object field to be measured here greatlyfluctuates, which means that the results are not always reliable.

DESCRIPTION OF THE INVENTION

It is therefore an object of the invention to produce an overview imageof a sample carrier in which the structure of the sample carrier itselfand possibly further structures, for example of the inscription on thesample carrier, of possible sample regions, of any immersion liquid thatmay be present, of the actual sample and/or of sample errors such as airbubbles, dirt etc., can be detected clearly and without errors. Comparedto normal recordings of a camera and even HDR images, the overview imageis intended to have increased contrast or an improved signal-to-noiseratio of the particular structures of interest, which is why theoverview image will be referred to as an overview contrast image below.The overview contrast image can be made available to the user for thepurposes of navigation, or it can be used to improve the automatedanalysis of samples and to reduce the susceptibility to errors, forexample by detecting sample errors.

This object is achieved by a method of the type described in theintroductory part, in which the overview contrast image is produced asfollows: A sample carrier—generally carrying at least one sample—isarranged at least partially in the focus of a detection optical unit andis illuminated in transmitted light with a two-dimensional, preferablyarray-type, illumination pattern. At least partially here means that thesample carrier or the sample does not need to be completely visible, butmeans in particular that the sample carrier and/or the samples can alsohave an extent along the optical axis that is greater than the depth offield of the detection optical unit used. A detection optical unit thatcan be used can be the optical unit of a microscope, although it ispreferably a camera having a camera objective lens, which makes itpossible to image a large object field onto a surface detector,preferably with a sufficiently high depth of field.

In this refinement of the method, it is necessary for producing theoverview contrast image to detect at least two overview raw images withdifferent illuminations of the sample carrier.

The overview raw images, that is to say unprocessed images, are thendetected for example by the camera by way of a surface detector—forexample a CMOS chip—registering the intensity pixel by pixel. Dependingon the type of illumination, the overview raw images can be recordedsuccessively or simultaneously, for example using a single camera.

A calculation algorithm that is used to calculate an overview contrastimage from the at least two overview raw images is chosen in dependenceon the type of the illumination and information that is to be extractedfrom the overview contrast image. The information can be, for example,the already mentioned structures of the sample carrier, of the sampleetc., and of the inscription on the sample carrier.

Likewise in dependence on the information that is to be extracted fromthe overview contrast image, an image evaluation calculation algorithmis chosen that is used to extract the information from the overviewcontrast image. Said information can then be used for example by theuser on the screen of a connected computer to initiate further steps aspart of the observation and analysis, for example to navigate on thesample, which is accomplished by the image being represented on a screenand the user choosing the sample region of interest for example by wayof a mouse click. On account of the image evaluation, the microscopesystem can then be automatically adjusted to that position. However, theinformation can also be used as part of an automated examination, forexample in high-throughput examinations, to exclude incorrectly filledwells of a multiwell carrier, such as a microtiter plate, with theresult that the microscope is not even adjusted to their positions.

Essential aspects relate to the use of a two-dimensional, in particulararray-type illumination and to the recording of the overview raw imageswith different illuminations. Two-dimensional illumination can beobtained in different ways, wherein preferably an array of illuminationelements of the same size is used to produce illumination patterns. Atany rate, the individual illumination elements must be distinguishablefrom one another in the at least two overview raw images, that is to saythey must be able to be represented separately from one another in theimage, even though they were not arranged at the focus. The illuminationelements can be LEDs, OLEDs, the ends of optical fibers, elements of anilluminated DMD (digital micromirror device) or of a different spatiallight modulator. They can be elements that actively or passively emitlight. The illumination elements can also be produced for example with alight source that emits light over a surface and upstream of which forexample a switchable filter structure is arranged, with which one ormore properties of the illumination elements—for example color,intensity or polarization—can be manipulated. With particular advantage,however, light-emitting diodes (LEDs) can be used, because these can bearranged in nested arrays of multicolor LEDs and also provide asufficiently high light output, and moreover, microscopes that use anLED array, that is to say LEDs arranged in the manner of a matrix or agrid (LED array microscopes, angular illumination microscopes—AIM), forillumination are already available. The LED arrays of such microscopescan be likewise used to produce two-dimensional illumination patterns.

The different illuminations with which the at least two overview rawimages are recorded can be realized in different ways. One simplepossibility is to use a static two-dimensional illumination pattern andto move the sample carrier laterally, that is to say perpendicularly tothe optical axis of the detection optical unit, relative to theillumination pattern between two recordings. This can be accomplishedeither by moving the illumination pattern itself—likewise arranged in aplane with the optical axis as a normal—or by moving the sample carrier.The sample carrier or the sample is here illuminated in transmittedlight, that is to say the sample carrier is located between theillumination elements of the illumination pattern and the detectionoptical unit, for example the camera.

In addition to spatially different illuminations, it is also possible touse illuminations that differ in terms of time, for example by recordinga plurality of overview raw images with different exposure times or withillumination of differing lengths with the same exposure time, with thesignal-to-noise ratio being less favorable in the latter case. Using theHDR (high dynamic range) method known in the prior art, an overviewcontrast image can be calculated from said overview raw images.

Another possibility is to produce spatially different illuminationsusing different illumination patterns, wherein the illumination patternsare preferably chosen in dependence on the information that is to beextracted. In principle, a multiplicity of illumination patterns thatcan be impressed on the array of illumination elements, for example, aresuitable. For example, different illumination patterns can be producedby driving the illumination elements individually or in groups andswitching them to produce different illumination patterns, wherein afirst portion of the illumination elements is switched to emit light andat least a second portion of the illumination elements is switched toemit no light or to emit light of a different color or light of adifferent polarization. If the at least second portion of theillumination elements does not emit light, each pattern includes exactlytwo parts, and the overview raw images are detected successively. If thesecond portion of the illumination elements emits lights of a differentcolor, the illumination elements can also be divided into more than twogroups and comprise a third or further parts that emit light ofrespectively different colors, wherein the colors differ from oneanother in pairs. A corresponding statement applies to thepolarizations. Upon illumination with light of different colors, theoverview raw images can be recorded simultaneously, provided that, onthe detection side, separation into the different color channels iseffected. For example, if the array of illumination elements comprisesLEDs in the three RGB primary colors red (R), green (G) and blue (B),and if the sensor of the camera has corresponding sensors assigned tosaid color channels, a separation is readily possible, and threeoverview raw images can be recorded at the same time. Similar is truefor polarized illumination, for example if an LED array is provided withpolarization filters of different polarizations and the polarizationdirections are likewise detected and used for separating the channels.

If the illumination elements are in the form of LEDs, and an LED isformed from three individual mutually adjacent sub-LEDs that each emitlight in different primary colors red, green and blue, the differentilluminations can also be set by illumination from different angles inthe primary colors. Here, too, the overview raw images can be detectedsimultaneously.

Another possibility is to produce explicitly different illuminationpatterns and to record the overview raw images successively withdifferent illumination patterns. This can be done in a first variant forexample by stochastically choosing the first portion of the illuminationelements for each illumination pattern, wherein the individualillumination elements can randomly be driven and switched to emit lightor to emit no light, wherein care should be taken to ensure by way ofboundary conditions the best possible equal distribution of illuminationelements that emit light and those that do not emit light. In a secondvariant, pulse-width-modulated illumination elements are used—this canbe realized particularly well with LEDs or OLEDs—wherein the pulse widthis selected to be longer than the integration time of a detector unitfor recording the overview raw images. In this case, the illuminationelements do not need to be driven individually.

Instead of stochastic illumination patterns, it is also possible toproduce illumination patterns in which the illumination elements exhibita regular distribution. For example, chessboard-type, cross-shaped orhalf pupil distributions can be used for the light-emitting illuminationelements. In particular in the case of chessboard-type distribution,there are two good possibilities: First, the second portion of theillumination elements may emit no light, in which case two overview rawimages are recorded successively with mutually complementaryillumination patterns. If the illumination is in the manner of achessboard, the patterns are inverted with respect to one another.Second, the two parts of the illumination elements can also emit lightof different colors or polarizations, in which case the overview rawimages can be recorded simultaneously in one image and subsequently beseparated into color channels or polarizations. In the case of achessboard-type illumination with two illumination patterns, the twopatterns are not only complementary, but also inverted with respect toone another. If a plurality of patterns—for example a singlelight-emitting illumination element that scans the array—are used, allpatterns together behave in a mutually complementary fashion, that is tosay, overall, they produce an array of only light-emitting illuminationelements. In the case of half pupil illumination, in each case two outof the four necessary illumination patterns are mutually complementary.

Finally, it is also possible to realize different illuminations byselecting from the array of illumination elements at least one sectionand producing the illumination pattern only in said section. Thedifferent illuminations are achieved by a scanning movement of the atleast one section on the array, wherein the illumination elementsoutside of the at least one section are switched to emit no light. Forexample, illumination with an individual LED can be used here, or asection of a chessboard-type illumination. If a sample carrier is large,it is possible here to choose a plurality of sections that are moved inparallel fashion; a combination with differently colored illuminationelements is likewise possible to produce a plurality of sections at thesame time.

After the overview raw images have been recorded, a calculationalgorithm that is used to calculate the overview contrast image from theat least two overview raw images is chosen in dependence on theinformation that is to be extracted from the overview contrast image.The choice of the calculation algorithm is preferably also made independence on the previous choice of the illumination method, i.e. isadapted thereto. The overview contrast image is preferably producedeither in dark-field mode or in bright-field mode, because these permitthe best contrast, but mixed modes are also possible. It is necessaryhere to ensure that the contrast is optimum for the structures ofinterest, for example depending on whether the cover slips, the sampleitself, or the inscription is/are to be represented with the highestpossible contrast. Under certain circumstances, overview contrast imagescan be produced from the overview raw images both in bright-field and indark-field mode, depending on the desired information. In addition tothe production of an overview contrast image in a dark-field orbright-field mode, other types of contrast can also be produced, forexample overview contrast images in a HDR mode, which contain dark-fieldand bright-field components.

In a first configuration, the calculation algorithm is based on apixel-wise projection, preferably a ranking projection or a projectionof statistical moments. For producing the overview contrast image from astack of at least two overview raw images, the overview raw images arecompared pixel by pixel, and the intensity value of one of the pixelsfor the corresponding position in the overview contrast image is chosenin accordance with a projection condition. The recorded image stack isconsequently subjected to calculation pixel by pixel, that is to sayeach pixel in the overview contrast image is influenced only by thepixels in the image stack that are located at the same image position inthe overview raw images. In the case of a ranking projection, thecorresponding pixel values of the overview raw images for an imageposition are sorted by intensities, and the value corresponding to thep-quantile is used for the overview contrast image, with p being aparameter that is to be specified by the user or is specified by thecalculation method. A special case is the minimum projection with p=0.0,in which the pixel with minimum intensity is chosen; other special casesare the maximum projection with p=1, in which the pixel with maximumintensity is chosen, or the median projection with p=0.5.

Depending on the illuminations with which the overview raw images wereproduced, overview contrast images can be produced in dark-fieldcontrast or in bright-field contrast with this type of calculation. Anoverview contrast image in bright-field mode can be produced for exampleif the illumination elements—in particular if these are in the form ofLEDs—are not overdriven and p=1, i.e. the maximum projection is chosen.However, the overview contrast image can be also produced in dark-fieldmode, for example if the brightest possible overview contrast image isproduced with the greatest possible p, wherein only pixels that have notbeen directly illuminated by an illumination element will be taken intoaccount in the calculation. For example, if two overview raw images areproduced with a chessboard-type distribution of the first portion of theillumination elements and a distribution that is complementary thereto,p=0 is selected and a minimum projection is performed. If, by contrast,the illumination pattern is produced only in a section of the array andthis section is moved in a scanning fashion on the array, each pixel inthe image is directly illuminated less often than overview raw imagesare recorded. For example, if four LEDs that are switched on in the formof a cross are used and 30 overview raw images are produced for ascanning chessboard pattern, each pixel is illuminated directly at mostfour times by an LED having a significantly larger diameter than apixel. The value is then p=((30−1)−4)/(30−1)=0.8621.

Alternatively, it is also possible to use an algorithm based on theprojection of statistical moments. In this case, each pixel in theoverview contrast image corresponds to a statistical moment, such as forexample the standard deviation of the intensity over the correspondingpixels of the overview raw images. In particular in combination with asequence of statistical LED patterns that are moved laterally relativeto the sample carrier, this offers good contrasts and maintains evensmall details, with the result that this calculation algorithm isparticularly suitable for example for detecting multiwell samplecarriers or chamber-slide sample carriers.

The advantages of the above-described projection methods as calculationalgorithms are that they are able to be parallelized very well andconsequently permit very fast calculation, and, in addition, owing tothe equal treatment of all pixels, no seam artefacts occur, as would bethe case for example with calculation algorithms based on segmentation,in which, in unfavorable cases, the boundaries of the structures seem todiscontinuously jump at the seams.

In another configuration, the calculation algorithm is based onmorphological operations with subsequent pixel-wise projection,preferably on a top-hat or black-hat transform with subsequentpixel-wise maximum projection. It is possible using the top-hattransform to highlight bright structures on a dark background, and theblack-hat transform can be used to highlight dark structures on a brightbackground. These calculation algorithms can be used to make inparticular glass edges, that is to say edges of sample carriers or coverslips, visible. Next, a pixel-wise maximum projection over the overviewraw images thus transformed is formed and the overview contrast image isproduced in this way. The advantages of this calculation algorithm arethat the information is acquired at the same time from bright-field anddark-field contrasts, and that it is likewise efficiently subjected to acalculation. However, as compared to a ranking projection, the contrastin the images that have been subjected to calculation is mostly lowerand is frequently visible only in the case of glass edges. In addition,strong background artefacts can be produced that must then be redressed.

In a further configuration, a calculation algorithm based onsegmentation is chosen, in which initially a determination is made foreach pixel of an overview raw image as to whether said pixel has beendirectly irradiated with light by an illumination element. These pixelsare then not taken into account for the production of an overviewcontrast image in the dark-field contrast mode. The overview contrastimage is produced using a projection method in this case, too. Theadvantage of this calculation algorithm as compared to the rankingprojection is that an explicit determination is made here as to whichpixel information from the overview raw images can be used. Adisadvantage is that, on account of the segmentation and the resultingunequal treatment of pixels, seams can form in the calculated contrastimage. In addition, the calculation cannot be performed as efficientlyas in the previously described calculation algorithms.

In one alternative configuration of the method operating in bright-fieldmode, the overview contrast image is not produced by calculation but isdetected directly, that is to say no overview raw images are detected,or, in other words, the overview raw image in this case is identical tothe overview contrast image. To this end, a diffusion screen is insertedinto the beam path between the array-type illumination pattern—which canbe made up of an array of illumination elements of preferably equal sizein this case, too—and the sample carrier. The diffusion screen, whichproduces diffuse illumination, which is advantageous for bright-fieldillumination, can also permanently remain in the beam path, provided itis switchable, and it is then switched on, i.e. switched to diffusion,only for the production of an overview contrast image in bright-fieldmode.

If the sample carrier is laterally moved relative to the illuminationpattern between two recordings, it is necessary to know for thecalculation how the sample or the illumination pattern in that casemoves in the image. To this end, the camera is calibrated relative tothe sample carrier or the illumination pattern in order to be able tomap—in the case of a movement of the sample carrier—the coordinates of astage, on which the sample carrier is held and which can be displaced tomove the sample carrier, onto image coordinates. A similar procedure canbe used for movable illumination patterns. In the case of a displaceablestage, a calibration pattern, for example a chessboard, is placed in thestage concretely for calibration purposes, as a result of which thecoordinate mapping can be estimated with sufficient accuracy.Alternatively, such calibration can also be dispensed with in thecalculation and the movement of the sample can be ascertained by way ofimage analysis or using a different measurement system.

In particular if the illumination pattern is stationary and the samplecarrier is moved between the recordings, an overview contrast image canbe generated after calibration, that is to say after a quantification ofthe actual movement of the sample carrier in relation to the image, evenfor larger sample carriers that do not entirely fit into the objectfield that is capturable by the camera or the detection optical unit, byinitially producing individual contrast images that each show differentregions of the sample carrier—and/or of the sample—and are produced fromcorresponding individual overview raw images. Said individual contrastimages are subsequently combined to form the overview contrast image,wherein the calibration is used to correctly locate connecting pointsfor joining them.

Notwithstanding the above, a calibration is also advantageouslyperformed using the recording and evaluation of a calibration patternfor correcting geometric distortions before the recording of overviewraw images. The calibration pattern is an object of known geometry andclearly detectable structure—for example the aforementioned chessboardpattern—which is placed at different positions in the image field of thecamera and is recorded with the camera. However, it is also possible touse the array of illumination elements, in particular if these are inthe form of LEDs, as a calibration pattern. Such calibrations are knownfrom the prior art.

As has already been indicated in connection with calculation algorithmsbased on segmentation, background artefacts may occur in the producedoverview contrast image depending on the calculation algorithm used.Such disturbing background signals are removed, i.e. by calculation,preferably after the production of the overview contrast image yetbefore the evaluation thereof using corresponding correction algorithms.If there is no lateral movement of the illumination pattern or thesample carrier, the artefacts typically form a periodic structure thatfollows the positions of the individual illumination elements. These canthen be removed using what is known as self-filtering. Additionalcorrection methods are known in the prior art that can be used to removethe occurring background artefacts or at least to reduce them, inparticular by recording or calculating a background image, which is thensubtracted from the overview contrast image, i.e. removed therefrom bycalculation. For example, a background image can be determined from thecalculation of the overview raw images by averaging out the foreground.A background image can also be determined from a recording without thesample carrier or with an empty sample carrier. By subjecting thebackground contrast image to a calculation, a background image can bedetermined for example by calculating the average pixel values in localregions around an illumination element that does not emit light by wayof analyzing all illumination elements in the overview contrast image.This is because the foreground structure is independent of the positionrelative to the illumination element and is averaged out, but thebackground structure is dependent on said position and is consequentlyintensified. This produces a background image via the position of theillumination elements that can subsequently be calculated out of theoverview contrast image. Another possibility for calculating outbackground artefacts is the use of a bandpass filter, possibly also incombination with non-linear filters.

In a final step, the background contrast images are finallyautomatically analyzed using an image processing algorithm that is to bechosen and the required information is extracted. The information thatis to be extracted comprises one or more of the following data: type ofthe sample carrier, inscription in the sample carrier, locations ofsamples or sample regions, cover slips, wells of a multiwell plate inthe image, information relating to an immersion liquid such as position,volume, shape, positions of artefacts, defective samples, air bubblesetc. This information can be reliably extracted only on account of thehigh contrast in the overview contrast images.

An image processing algorithm that can be chosen is for example analgorithm based on the principle of machine learning, in particular adeep learning algorithm, which is preferably trained on the basis ofoverview contrast images with known information.

For automated extraction of the aforementioned information from theoverview contrast images, it is expedient to use methods from the fieldof machine learning. In this case, an annotated training samplecomprising a quantity of contrast images that are to be analyzed isprovided for the respective task, including for example imageclassification, segmentation, localization, detection. Each contrastimage is here assigned a desirable output corresponding to the task, aswill also be explained below with reference to examples. Usingtechniques of machine learning, it is then possible for a model to beautomatically adapted in a learning step such that the desirable andcorrect outputs are produced even for non-viewed, i.e. new images.

Possible algorithms based on techniques of machine learning will beoutlined by way of example below. Alternatively, methods fromtraditional image and signal processing can be used, but algorithmsbased on machine learning, and in particular based on deep learning,offer significant advantages for example in terms of quality,robustness, flexibility, generalizability, and development andmaintenance complexity.

A deep learning algorithm based on a convolutional neural network can beadvantageously used to identify the type of a sample carrier, forexample whether it is for example a multiwell sample carrier, a Petridish, a chamber slide etc. The task is thus one from the field of imageclassification, and the algorithm uses an image as input and outputs aclass. The training sample in this case comprises contrast images, andeach contrast image is assigned one of the suitable sample carriertypes, a first contrast image is assigned the type “Petri dish,” asecond contrast image is assigned the type “multiwell plate,” a thirdcontrast image is assigned the type “chamber slide,” a fourth contrastimage is assigned the type “slide,” and so on.

A convolutional neural network (CNN) consists of different layers, forexample convolutional layers, pooling layers, non-linear layers, etc.,the arrangement of which is specified in the network architecture. Thearchitecture used for the image classification follows a certain basicstructure, but is in principle flexible. Each element of the networkreceives an input and calculates an output. In addition, some elementsof the network have free parameters that determine the calculation ofthe output from the input. A three-dimensional number array, i.e. acolor image having in each case three color values per pixel, is inputto the first layer as an input of the network. The last layer thenoutputs a probability distribution over all possible sample carriertypes as the output of the network—for example the output for anoverview contrast image is thus: “slide” 87%, “multiwell plate” 1%,“Petri dish” 2%, “chamber slide” 10%. Optionally, a rejectionclass—which provides for example the values“unknown”/“invalid”/“empty”—can also be integrated. On account of thetraining process, the free parameters of the network are adapted on thebasis of the provided training data such that the outputs of the modelmatch the expected outputs as closely as possible.

The training can also use a model that has already been trained forother data as a starting point in the sense of a fine adjustment, whichoffers advantages in terms of quality, time requirements, and dataexpenditure.

As an alternative to CNNs or methods derived therefrom or relatedmethods of deep learning, the image classification can also be performedusing traditional methods of machine learning, which typically comprisetwo steps: (I) feature extraction, and (ii) classification. In the caseof the feature extraction of step (i), the overview contrast image istransformed, using a predefined algorithm, into an alternativerepresentation—typically into a compact or sparse vector. Simpleexamples here are for example local gradient histograms (histograms oforiented gradients, HoG). In the classification of step (ii), each ofthese feature vectors is then assigned a class using a classifier. Oneexample of a classifier is a support vector machine (SVM). The freeparameters of the classifier are here likewise adapted in the trainingstep such that the actual outputs match the desirable outputs as closelyas possible.

A hybrid approach between traditional machine learning and deep learningis based on the use of a CNN for the feature extraction in step (i). Inthis case, a CNN that has been trained for different data is cut off ata specific layer, and the activation of the network is used as a featurevector.

The localization of structures of the sample carrier, for example coverslips in the case of slides and Petri dishes or chambers of chamberslides or multiwell plates, on the one hand, and of structures of thesample or sample regions on the other can be considered a problem ofsemantic segmentation, that is to say, for an overview contrast image asthe input image, an image in which each pixel of the input image isassigned a class (e.g. “background,” “cover slip,” “dirt,” “sample,” . .. ) is to be returned. This can preferably likewise be implemented withnetworks from the field deep learning, for example using fullyconvolutional networks (FCNs) that are based on CNNs.

Like CNNs, FCNs typically expect as an input a three-dimensional numberarray, i.e. a colored overview contrast image, but output an array inwhich each pixel of the input image is assigned a probability relatingto it being part of each of the occurring classes. The training samplein this case comprises contrast images, and each contrast image isassigned an array (graylevel image) of the same size, in which eachpixel is in turn assigned a class—coded via a gray level. Trainingproceeds similar to in the case of CNNs. It is possible here to usedifferent FCNs for the localization of sample carriers and for thelocalization of sample regions, but the evaluation is in particularpossible using a single FCN, which images or contains for example both“coverslip” and “sample” as classes.

It goes without saying that the aforementioned features and those yet tobe explained below can be used not only in the combinations specifiedbut also in other combinations or on their own, without departing fromthe scope of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is explained in even greater detail below for example withreference to the accompanying drawings, which also disclose featuresessential to the invention. In the figures:

FIG. 1 shows an arrangement for performing a method for producing andanalyzing an overview contrast image,

FIG. 2 shows the construction of a microscope that is suitable therefor,

FIG. 3 shows a detail of an illumination,

FIG. 4 shows the production of stochastic illumination patterns,

FIG. 5 shows two complementary chessboard-type illuminations,

FIG. 6 shows a chessboard-type illumination with different colors,

FIG. 7 shows a cross-shaped distribution of the illumination elements,

FIG. 8 shows a half pupil distribution of the illumination elements,

FIG. 9 shows the scanning movement of a section with an illuminationpattern,

FIGS. 10-12 show the production of overview contrast images fromoverview raw images with static illumination patterns.

DETAILED DESCRIPTION OF THE DRAWINGS

To begin with, FIG. 1 outlines an arrangement with which an overviewcontrast image of a sample carrier 1 and/or samples arranged on thesample carrier 1 can be produced. The sample carrier 1 is here arrangedat least partially in the focus of a detection optical unit 2 and isilluminated in transmitted light with a two-dimensional, array-typeillumination pattern 3. In order to produce the overview contrast image,at least two overview raw images are initially detected with differentilluminations of the sample carrier 1. To this end, by way of example asurface detector 4 is used here, onto which the detection optical unit 2images. The detection optical unit 2 can be a microscope objectivehaving a small magnification, although it is preferably the objectivelens of a camera, which is able to image a larger region of an objectfield, which then ideally covers the entire sample carrier 1 in anoverview. Accordingly, the surface detector 4 in this case is the sensorof the camera, for example a CMOS chip. Depending on the configurationand in particular in dependence on the selection of the illumination,the surface detector 4 registers only the intensities—for example in thecase of white illumination—or the intensities are separated intodifferent color channels, for example red (R), green (G) and blue (B).Similar to different colors, it is also possible to take into accountdifferent polarizations in the illumination using appropriate sensorsthat also register the polarization and to use said polarizations forseparation into different channels.

Depending on the type of the pattern and on the type of theillumination, the overview raw images are recorded either at the sametime or in succession, wherein for each pixel the correspondingintensity values are registered. The overview raw images are thensupplied to a calculation unit 5 for subjecting them to a calculation toobtain an overview contrast image. In the calculation unit 5, acalculation algorithm that is used to calculate an overview contrastimage from the at least two overview raw images is chosen in dependenceon information that is to be extracted from the overview contrast image,and optionally also in dependence on the illumination. The overviewcontrast image is then supplied to an image evaluation unit 6, in whichan image evaluation algorithm that is used to finally extract theinformation is selected in dependence on the information that is to beextracted from the overview contrast image. The information istransmitted to a control unit 7, which correspondingly processes itfurther and excludes from the microscopic analysis, for example in ahigh throughput method, such multiwells in which the evaluation of theoverview contrast image has indicated that said multi-wells have notbeen correctly filled, for example contain defective samples or airbubbles, etc. The overview contrast image can of course also berepresented to a user on a screen, which is connected to the imageevaluation unit 6 or the control unit 7 and can be part of said units,with the result that a user can manually perform corresponding settings.The calculation unit 5, image evaluation unit 6 and control unit 7 cantogether be integrated in a PC as hardware and/or software.

As has already been indicated in connection with the descriptionrelating to FIG. 1, the method can also be readily performed withexisting microscopes. Particularly suitable are microscopes that use anLED array for the illumination, in which case the illumination pattern 3is produced by the LED of said array. Such a microscope, which uses forexample angular illumination microscopy (AIM), is illustrated by way ofexample in FIG. 2. The sample carrier 1 is illuminated here via an LEDarray 8, and the detection optical unit 2 here comprises by way ofexample two lens elements, between which a deflection mirror 9 forfolding the optical axis is arranged. A beam splitter 10 is used toguide some of the light onto a surface detector 4, while a differentpart of the light is directed, via a lens element 11, onto an eyepiece12, such that the overview raw image can also be viewed by an observer.The camera and illumination can be positioned particularly well on aninverse microscope stand, for example by way of the LED array 8, withwhich the illumination pattern 3 is produced, being arranged above anarm, which is prepared for the transmitted-light illumination, and thecamera being placed below the sample for example on the objectiveturret.

For producing illumination patterns 3, an array having illuminationelements that preferably have the same size is preferably used.Illumination elements that can be used are for example LEDs, OLEDs,optical fibers, i.e. the ends or exit faces thereof as active lightsources, or elements of an illuminated DMD (digital micromirror device)or of a different spatial light modulator as passive illuminationelements. If the following text refers to LEDs for example as lightsources, this is done only for illustrative purposes and does notexplicitly exclude the use of the other possible arrays of illuminationelements.

The overview raw images are recorded by way of a camera with the surfacedetector 4; the objective lens of the camera is focused, as shown inFIG. 1, at the sample carrier 1 and directed at the illumination pattern3 or the illumination behind the sample carrier 1, which is not situatedin the focus. It is possible here to use a camera having an objectivelens that has a large object field and is not telecentric. No additionaloptical elements need to be placed between the sample carrier 1 and theillumination pattern 3, which can be configured for example as an LEDarray, to manipulate the illumination. Typically, the distances betweenthe detection optical unit 2 or the surface detector 4 and the samplecarrier 1 and between the sample carrier 1 and the illumination pattern3 can in each case be selected to range between 0.5 cm and 10 cm, butthey can in particular be even larger, possibly to capture the entiresample carrier 1.

In principle, the distances can also be selected freely, as long asvarious conditions have been met: (i) the sample carrier 1 must belocated in the focal plane of the detection optical unit 2; (ii) thestructures of the sample carrier 1 to be analyzed—for example edges ofcover slips—can still be resolved by the camera; (iii) the structuresproduced by the illumination pattern 3 must be discernible on the imagethat is registered by the surface detector 4, i.e. individualillumination elements must be distinguishable and advantageously coverthe entire structure that is to be analyzed, which can be influenced bya corresponding choice of the size of the array of illuminationelements, the size of the illumination elements, and the spacingthereof, which is why for example an array of LEDs is highly suitablefor larger structures such as sample carriers. If they do not completelycover the structure, a combined overview contrast image can be producedwith corresponding calibration.

The illumination will be explained in more detail below with referenceto FIG. 3, in which LEDs are used as examples of illumination elements.Of the illumination pattern 3, only one LED 13, which is arranged behindthe sample carrier 1, was depicted here as an example. The distances arechosen at random, and the illumination pattern 3 can in fact be arrangedat an even greater distance. However, arranging it directly behind thesample carrier 1 permits arrangement in the vicinity of the focal plane,which means that the resolution of the individual illumination elementsis improved.

The recording is taken with a detection optical unit 2, which is nottelecentric. Each switched-on LED 13 acts either as bright-field ordark-field illumination, depending on the field region of the sample.For a first field region 14, the LED 13 is arranged directly behind thesample or the sample carrier, where a transmitted-light componentdominates, such that the LED 13 acts as bright-field illumination forthis first field region 14 and produces a corresponding bright-fieldcontrast. For a second field region 15 next to the LED 13, by contrast,the LED 13 acts as dark-field illumination and can be used to produce adark-field contrast. If the intention is to produce an overview contrastimage in bright-field mode, it is possible to optionally insert adiffusion screen between the array of illumination elements and thesample carrier 1, because diffuse light sources are advantageous forbright-field contrasts, although not for dark-field contrasts. Thediffusion screen can be inserted in each case, but can also bepermanently positioned in the beam path, and can be switchable, with theresult that light diffusion occurs only if the diffusion screen isswitched on. By producing overview raw images with differentilluminations, which can be realized in particular by differentillumination patterns 3, the bright-field and/or dark-field informationof the sample carrier 1 and also of the sample itself can be extractedand be represented in an overview contrast image.

The different illuminations are selected in dependence on theinformation that is to be extracted. This information generally includesthe type of the sample carrier, for example whether the latter is anormal slide in the sense of a small glass plate, or a simple Petridish, or a multiwell sample carrier having a multiplicity of open wells,or a sample carrier having different, closed sample chambers (chamberslide), which are therefore covered by a cover slip. Frequently, asample number is indicated on the sample carrier 1, for example by wayof a handwritten inscription, but more frequently as a code with abarcode or QR code, which can be interpreted in conjunction with acorresponding database. In particular when using sample carriers havinga plurality of sample chamber or wells, the intention is to determinepossible sample regions. In addition, the samples must be able to beidentified and errors or faults, such as air bubbles, contamination orempty sample chambers, must be able to be detected. It is furthermorethe intention to be able to detect the presence of immersion liquid andthe form, volume, and position thereof.

Once the desired information has been automatically extracted from theoverview contrast image using an algorithm for image processing, thefollowing steps can be performed automatically or semiautomatically,depending on the task. An example to be mentioned is the choice of whichwell in a multiwell plate is to be examined, which can be done eitherautomatically or only by the PC, without a user being required to onceagain look through the eyepiece, but in which it is likewise possible torepresent an overview image of the complete sample carrier 1.

Different illuminations and calculation algorithms that are suitableherefor will be described below with reference to FIGS. 4-12. Forillustrative purposes, an LED array is always used, although the use ofdifferent illumination elements, such as were mentioned above by way ofexample, is likewise readily possible.

A first possibility is to produce the different illuminations usingdifferent illumination patterns 3, which are chosen in dependence on theinformation that is to be extracted. Such illumination patterns areillustrated in FIGS. 4-9.

For example, different illumination patterns 3 can be producedstochastically. This is illustrated in FIG. 4, where the six frames showdifferent illumination patterns, in which the LEDs 16 of an LED array 17emit white light and are stochastically switched on or off. In theswitched-on state, they are shown as small circles, and in theswitched-off state, they have been omitted to clearly mark the differentillumination patterns.

A simple possibility for producing stochastic illumination patterns 3 isto use pulse-width-modulated illumination elements having a pulse widththat is selected to be longer than the integration time of a detectorunit for recording the overview raw images, wherein this selection canalso be made by way of specifying an integration time. During theintegration time of the camera, some of the LEDs are then switched onand others are switched off, because the pulse-width modulation betweenthe LEDs 16 is not synchronized. The LEDs 16 of the LED array 17 in thiscase do not need to be individually drivable or switchable.

It is of course also possible for the illumination elements to be drivenand switched as different illumination patterns individually or ingroups. A first portion of the illumination elements are here switchedto emit light and at least one second portion of the illuminationelements is switched to emit no light or to emit light of a differentcolor or to emit light of a different polarization. In the case of theproduction of stochastic illumination patterns in FIG. 4, the firstportion of the illumination elements is stochastically chosen for eachillumination pattern 3. The second portion of the illumination elementsdoes not emit light. In order to be able to produce a high-qualityoverview contrast image, relatively many overview raw images arerequired, with the result that generally several seconds are required torecord the overview raw images. This period can be shortened if, insteadof switching off the second portion of the illumination elements, itemits light of a different color, with the result that two overview rawimages are recorded at the same time, which are subsequently separatedby color.

Overview contrast images can be produced both in a dark-field mode andin a bright-field mode, depending on the calculation algorithm, which inthis case can be based for example on a top-hat transform for abright-field contrast image or on a black-hat transform for a dark-fieldcontrast image, in each case with subsequent, pixel-wise maximumprojection, wherein both transforms can be applied equally to theoverview raw images such that it is possible to produce an overviewcontrast image both in bright-field mode and in dark-field mode. Withthis type of illumination, glass edges, that is to say the peripheriesof the sample carriers 1, or cover slips can be rendered very visible,they exhibit high contrast as compared to the actual sample.

While relatively many images—typically between 30 and 50—need to berecorded if stochastic illumination patterns are used to obtainsatisfactory contrasts in the overview contrast image, otherillumination patterns work with far fewer images. Such patterns areshown in FIGS. 5 and 6, which are chessboard-type patterns. Illuminationelements that are switched on and off—LEDs 16—here have achessboard-type distribution, in which case the first portion of theillumination elements which are switched on for example corresponds tothe white fields of the chessboard, and the switched-off second portionof the LEDs 16 corresponds to the black fields. Two overview raw imagesare required, which are produced with mutually complementary, i.e.inverted illumination patterns. These two chessboard patterns areillustrated in FIG. 5. The first portion of the illumination elements isformed by the switched-on LEDs 16, and only every other LED 16 in everyrow and every column is switched on. In the right-hand illuminationpattern of the two illumination patterns, the LEDs 16 that were switchedoff in the image on the left, are switched on, and vice versa. Theillumination pattern can here extend over the entire LED array 17 orover only a region of interest of the sample carrier 1 to reduce theoverall quantity of light and so as not to unnecessarily load thesample. The chessboard-type illumination can be used in particular forusing an overview contrast image in dark-field mode; the calculationalgorithm used here is in particular a ranking projection algorithm,based on pixel-wise projection, in the minimum projection. Only twooverview raw images are required, and the method offers very goodcontrast, both for the sample carrier 1 and sample regions—for examplethe glass edges of cover slips and for the sample itself.

When using illumination patterns as shown in FIG. 5, LED arrays 17 withsingle-color LEDs can be used, which emit for example white light orlight in one of the primary colors R, G, B; for the detection, a surfacedetector that registers multi-colored or monochromatic light can beused.

If the first portion of the illumination elements emits light and thesecond portion of the illumination elements does not emit light, twooverview raw images that must be recorded successively are required, inthe case shown in FIG. 5, to produce an overview contrast image. Theoverview raw images, however, can also be recorded simultaneously with acamera and subsequently separated when all portions of the illuminationelements emit light of polarizations that differ from one another inpairs. To produce the chessboard-type illumination pattern, an LED array17, which is provided with complementary polarization filters, whichalternate in rows and columns, depending on the pattern, can be usedhere. The polarization filters can also be switchable. In this way, thetwo overview raw images can be produced with one recording and only needto be separated subsequently, to which end the polarization mustlikewise be detected.

A further possibility is that all portions of the illumination elementsemit light of colors that differ from one another in pairs, that is tosay, for example in the case of four portions of illumination elements,that each of the portions emits light of a different color. This isexplained in FIG. 6 again on the basis of a chessboard-type illuminationpattern. The first portion of the illumination elements here comprisesblue LEDs 18, that is to say LEDs that emit light in the blue wavelengthrange, while the second portion of the illumination elements comprisesred LEDs 19, that is to say LEDs emitting light in the red wavelengthrange. The two grids are nested within one another, with the result thata red/blue chessboard is presented on the LED array 17. The samplecarrier 1 is illuminated with this illumination pattern, and a recordingthat already comprises both necessary overview raw images is taken. Theindividual overview raw images are obtained by separating the recordingby color channels. Depending on the configuration of the camera, that isto say on the number of the color channels and of the LEDs 13 of the LEDarray 17, it is also possible for three or more patterns to be nestedinside one another, with which the sample or the sample carrier 1 isilluminated at the same time. Ideally, the LEDs 16 of the LED array 17and the color channels of the camera that is used for the recording arematched to one another; without further measures, generally the threeprimary color channels red, green and blue are available, because evenLEDs emitting white light are made up of red, green and blue sub-LEDs.

Instead of an illumination pattern in the form of a chessboard,different illumination patterns can also be used, in which the firstportion of the illumination elements—and correspondingly the second andpossibly further portions—has a regular distribution as compared to thestochastic distribution. FIG. 7 shows such an example, a cross pattern,as it is called, in which four different illumination patterns areproduced and correspondingly four overview raw images are required. Thecontrast is here slightly greater than in chessboard-type illuminationpatterns. Four individual overview raw images are likewise needed whenusing half pupil patterns, as are shown in FIG. 8. For the production ofan overview raw image, the LED array 17 is divided into two halves, inwhich case the first portion of the illumination elements is located inone half and the second portion of the illumination elements, which isswitched-off, is located in the other half. The second overview rawimage is recorded with a distribution that is complementary thereto,that is to say if first the first portion of the illumination elementsfills out the left half on the LED array 17, then it will fill out theright half for the second overview raw image. Two further overview rawimages are produced by dividing the LED array into an upper and a lowerhalf, that is to say with a direction of division that is perpendicularto the first direction of division. In the case of transparent samplecarriers with vertical elements, in particular in the case of what areknown as chamber slides or transparent multiwell plates, it is possiblein this way to achieve a high contrast. The overview contrast image ispreferably produced in dark-field mode, to which end a calculationalgorithm that is based on pixel-wise projection is used, preferably aranking projection algorithm. Here, the overview raw images are comparedpixel by pixel, and the intensity value of one of the pixels is selectedfor the corresponding position in the overview contrast image inaccordance with a projection condition.

In a further configuration of the method, advantage is taken of the factthat each LED that emits white light is formed from three individual,mutually adjacent sub-LEDs, which in each case emit light in thedifferent primary colors red, green and blue. It is possible in thiscase to set different illuminations—the illumination patterns can herebe identical—by the illumination from different angles in the primarycolors. In this case, a calculation algorithm with which an overviewcontrast image in the bright-field mode is produced is chosen.

A further configuration of the method involves producing theillumination pattern 3 only in at least one section of the array ofillumination elements. The different illuminations are then produced byscanning movement of the at least one section on the array. Illuminationelements outside the at least one section are switched here such thatthey do not emit light. This is illustrated in FIG. 9 with the exampleof the chessboard pattern, of which a small section of four LEDs 16 waschosen here, which scans one row in a sequence of images or illuminationpatterns and is then moved, row by row, over the LED array 17. Ascompared to the chessboard pattern described in connection with FIG. 5,a greater number of overview raw images are required here, and thecontrast is comparable in terms of quality. However, the quantity oflight per time that is emitted by the LED array 17 is advantageouslysignificantly lower than when using both full patterns. The backgroundbrightness is therefore reduced, and fewer disturbing reflections occurin the overview raw images. The time required for the recording of theoverview raw images can be reduced by, for example, moving regions whichare spatially remote in the image of the sample carrier 1 sections withillumination patterns at the same time, and/or by producing illuminationpatterns in different colors, which are registered separately, in thesection that is to be moved.

In the section that is moved over the LED array 17, it is also possiblefor other patterns to be produced, for example it is possible for allLEDs except for one to be switched on, with the result that the sectioncomprises only one—switched-off—LED, and this section is then moved.Another possibility is to switch on only one LED and to leave all theothers switched-off, and to move this section over the array and in theprocess record the overview raw images.

A suitable calculation algorithm here is in particular a rankingprojection algorithm, in particular also in the minimum projection, withthe result that an overview contrast image in dark-field mode isobtained.

It may generally be necessary to overdrive the bright-field region onthe camera to obtain a good dark-field signal for dark-field contrasts.For a subsequent bright-field recording, it may then be necessary toperform a further recording without an overdriven bright-field region.

Another configuration of the method is lastly explained below withreference to FIGS. 10-12. Here, different illuminations are not producedby way of different illumination patterns, but by way of a lateralmovement of the sample carrier 1 relative to the illumination pattern 3between two recordings. With respect to FIG. 1, this corresponds to amovement that is perpendicular to the optical axis. It is possible hereeither to move the sample carrier 1 relative to the illumination pattern3 or the other way round, but it is also possible for both to be movedin relation to one another. Typically, movement of the sample carrier 1alone is easier to realize because the sample carrier is typicallymounted on a stage that is displaceable in all three spatial directions.The LEDs 16 of the LED array 17 are switched in a fixed pattern, forexample in a regular grid. The illumination pattern 3 and/or the samplecarrier 1 is moved between two overview raw images in a plane that isorthogonal to the optical axis of the camera. In FIGS. 10-12, differentsample carriers are illustrated in each case in four different positionsof the illumination pattern 3 relative to the sample carrier 1, which isrealized with the LED array 17 and the LEDs 16. The sample carrier inFIG. 10 used is a slide 20 with cover slips 21, which are to be madevisible; in FIG. 11, it is a multiwell plate 22, the wells 23 of whichare to be made visible; and in FIG. 12, the sample carrier is a chamberslide 24, the chambers 25 of which are to be made visible. The overviewcontrast images in all cases are illustrated on the right-hand side ofthe figures. Overview contrast images can here be produced both inbright-field mode and in dark-field mode. For an overview contrast imagein the dark-field made, the calculation algorithm used is the minimumprojection as a special case of the ranking projection, and in the caseof an overview contrast image in the bright-field mode, a maximumprojection can be used. To compensate for brightness differences, it ispossible here, as also in all other cases where it is deemed necessary,to perform a shading correction after the calculation. In the case of acalculation that is based on segmentation, the shading correction canalso be performed before the overview raw images are subjected to thecalculation.

On account of the relative movement between the recordings of theoverview raw images, it is necessary to know for the correct applicationof the calculation algorithm how the sample carrier or the illuminationpattern 1 moves in the image. To this end, it is necessary to calibratethe camera or the detection optical unit 2 relative to the samplecarrier 1 or to a stage on which it is mounted so as to be able to mapthe sample carrier coordinates onto image coordinates, and vice versa.To this end, initially a calibration pattern is used instead of thesample carrier at the same position or clamped onto the stage. In thisway, it is possible to estimate such a mapping—a homography, that is tosay a mapping of a two-dimensional plane onto a two-dimensional plane inspace. It is of course also possible to dispense with a calibration ifthe relative movement can be ascertained by an image analysis or using aseparate measurement system, or it is possible to perform calibration inadvance based on objective parameters and distances.

The overview contrast images that have been determined with statisticalillumination patterns provide the best contrast in terms of quality, inparticular when using LEDs, because the switched-off LEDs in the case ofthe dynamic patterns, that is to say in which the patterns change, canprovide quite a strong background signal on account of back-reflectionsat the sample carrier. Using corresponding image processing algorithms,for example with deep learning algorithms, these artefacts can, however,be eliminated in the evaluation, that is to say they will not be takeninto account.

One further possibility for using statistical patterns to producedifferent illuminations, without laterally moving the sample or thesample carrier 1 relative to the illumination pattern, is to combine anoverview contrast image from a plurality of recordings that were takenwith different exposures, in the manner of a HDR (high dynamic range)recording. It is possible to combine an overview contrast image as a HDRimage from for example three overview raw images which are recorded withdifferent exposures.

It is possible here when calculating the overview contrast image toadditionally take into account the position of the illumination elementswith respect to the sample or the sample carrier, as explained inconnection with FIG. 3. If the illumination element directly illuminatethe sample carrier or the sample, bright-field information is used, andif not, dark-field information. In this way, the overview contrast imageis a mixture of bright-field and dark-field contrasts.

The calibration pattern described above in connection with thecalibration of the relative movement can additionally be used to effecta correction of geometric distortions in the image, applied to eachoverview contrast image. In addition, it is also possible to eliminatebackground artefacts by calculation.

After the production of the overview contrast images, they areautomatically analyzed using an image evaluation algorithm, preferablywith an algorithm based on deep learning using neural networks. Forexample, the type of the sample carrier is identified, the samplecarrier can also be located in the image. If the sample carrier carriesan inscription, said inscription can likewise be determined from thecontrast image. The same is true for the sample or regions on thecarrier, such as wells that can contain samples. By detecting airbubbles or other artefacts by way of corresponding image evaluationalgorithms, it is possible in particular in the case of sample carriersthat include a plurality of samples in separate containers to reduce theexamination time taken for the sample carrier if such artefacts arepresent there. Finally, it is possible using the image evaluation of theoverview contrast image in the case of an immersion liquid to alsodetect the volume and the form of an immersion drop, and it is alsopossible to draw conclusions relating to contamination of the immersionliquid.

This information can preferably be displayed to an observer or user onthe PC by way of graphic means, such that the user can adapt his furtherprocedure to the results of the analysis of the overview contrast image.Although it may be sufficient under certain circumstances in the case ofoperation by a user to present the overview contrast image to said useralone, it is also possible to use the information obtained using theimage evaluation in particular for automated control of sampleexaminations with the microscope used. The overview contrast imageprovided to the user can use the control for navigating on the sample toprepare the further examinations. The information of the overviewcontrast image that is extracted using the image processing algorithmscan, however, also make possible a robust subsequent processing whichidentifies and localizes for example automatically relevant structuresof the sample carrier—such as the glass slides—or relevant structures ofthe samples on the sample carrier—such as tissue sections, organisms, orcells—for example to set a fully automated coarse positioning of thesample in all three spatial directions. Finally, the extracted imageinformation also permits a more robust, faster and more efficientautomated microscopy—such as high-throughput microscopy—with smallerdata volumes and shorter recording times with automatic exclusion oferror sources.

LIST OF REFERENCE SIGNS

-   -   1 Sample carrier    -   2 Detection optics    -   3 Illumination pattern    -   4 Surface detector    -   5 Calculation unit    -   6 Image evaluation unit    -   7 Control unit    -   8 LED array    -   9 Deflection mirror    -   10 Beam splitter    -   11 Lens element    -   12 Eyepiece    -   13 LED    -   14 First field region    -   15 Second field region    -   16 LED    -   17 LED array    -   18 Blue LED    -   19 Red LED    -   20 Slide    -   21 Cover slip    -   22 Multiwell plate    -   23 Well    -   24 Chamber slide    -   25 Chamber

The invention claimed is:
 1. A method for producing and analyzing anoverview contrast image of a sample carrier or of samples arranged onsaid sample carrier or of both the sample carrier and the samplesarranged on said sample carrier, comprising: illuminating the samplecarrier, which is arranged at least partially in a focus of a detectionoptical unit, in transmitted light with a two-dimensional, array-typeillumination pattern, detecting at least two overview raw images withdifferent illuminations of the sample carrier, choosing a calculationalgorithm that is used to calculate the overview contrast image with anincreased contrast or an improved signal-to-noise ratio from the atleast two overview raw images in dependence on information that is to beextracted from the overview contrast image, and choosing an imageevaluation algorithm that is used to extract the information from theoverview contrast image in dependence on information that is to beextracted from the overview contrast image.
 2. The method as claimed inclaim 1, further comprising producing illumination patterns using anarray having same-sized illumination elements, wherein the individualillumination elements are in each case distinguishable from one anotherin the at least two overview raw images.
 3. The method as claimed inclaim 2, wherein the illumination elements used are light emittingdiodes (LED), organic light emitting diodes OLED), optical fibers,elements of an illuminated digital micromirror device (DMD) or of aspatial light modulator (SLM).
 4. The method as claimed in claim 1,further comprising-producing the different illuminations by laterallymoving the sample carrier relative to the illumination pattern betweentwo recordings or by using different exposure times for the detection.5. The method as claimed in claim 4, further comprising combining theindividual contrast images that each show different regions of thesample carrier or the sample or of both the sample carrier and thesample to form the overview contrast image.
 6. The method as claimed inclaim 2, further comprising producing the different illuminations usingdifferent illumination patterns, which are chosen in dependence on theinformation that is to be extracted.
 7. The method as claimed in claim6, wherein producing different illumination patterns includes drivingthe illumination elements individually or in groups, and switching toproduce different illumination patterns, wherein a first portion of theillumination elements is switched to emit light and at least a secondportion of the illumination elements is switched to emit no light or toemit light of a different color or light of a different polarization. 8.The method as claimed in claim 7, wherein the different illuminationpatterns are produced by: stochastically choosing the first portion ofthe illumination elements for each illumination pattern, orstochastically choosing the first portion of the illumination elementshaving a chessboard-type or a different regular distribution, thedifferent regular distribution being a cross-shaped distribution or ahalf pupil distribution in the array.
 9. The method as claimed in claim8, comprising stochastically choosing the first portion of theillumination elements having a chessboard-type or a different regulardistribution, in which the second portion of the illumination elementsdoes not emit light, and at least two overview raw images are recordedwith mutually complementary illumination patterns.
 10. The method asclaimed in claim 8, comprising stochastically choosing the first portionof the illumination elements having a chessboard-type or a differentregular distribution, in which all portions of the illumination elementsemit light of colors or polarizations that differ from one another inpairs, further comprising recording simultaneously in an image a numberof overview raw images, which corresponds to the number of portions, andsubsequently separating the image by color channels or polarization. 11.The method as claimed in claim 6, wherein LEDs or OLEDs are used as theillumination elements, and the method further comprises producingstochastic illumination patterns by using pulse-width-modulatedillumination elements having a pulse width that is selected to be longerthan an integration time of a detector unit for recording the overviewraw images.
 12. The method as claimed in claim 6, wherein theillumination pattern is produced only in at least one section of thearray of illumination elements, and the different illuminations areproduced by scanning movement of the at least one section on the array,wherein the illumination elements outside the at least one section areswitched to emit no light.
 13. The method as claimed in claim 6, whereinthe illumination elements are in the form of LEDs, wherein an LED isformed from three individual mutually adjacent sub-LEDs that each emitlight in different primary colors red, green and blue, and differentilluminations are set by the illumination from different angles in theprimary colors.
 14. The method as claimed in claim 1, further comprisingchoosing a calculation algorithm for producing an overview contrastimage in a dark-field or bright-field mode in dependence on selection ofan illumination method.
 15. The method as claimed in claim 14, wherein:the calculation algorithm is based on pixel-wise projection, includingon a ranking projection or on a projection of statistical moments,wherein, for producing the overview contrast image from a stack of atleast two overview raw images, the overview raw images are comparedpixel by pixel, and the intensity value of one of the pixels for thecorresponding position in the overview contrast image is chosen inaccordance with a projection condition, or the calculation algorithm isbased on morphological operations with subsequent pixel-wise projection,including on a top-hat or black-hat transform with subsequent pixel-wisemaximum projection, or a calculation algorithm based on segmentation isselected, in which initially a determination is made for each pixel ofan overview raw image as to whether it was irradiated directly withlight by an illumination element, and said pixels are not taken intoaccount for producing an overview contrast image in dark-field contrastmode.
 16. The method as claimed in claim 2, further comprisingperforming a calibration for correcting geometric distortions using arecording and evaluation of a calibration pattern before the recordingof overview raw images, wherein the calibration pattern is produced withthe illumination elements of the array.
 17. The method as claimed inclaim 1, removing disturbing background signals from the overviewcontrast image before the evaluation.
 18. The method as claimed in claim1, further comprising: introducing a diffusion screen between thearray-type illumination pattern and the sample carrier, or switching aswitchable diffusion screen to a diffusing mode, wherein the overviewcontrast image is produced in a bright-field mode.
 19. The method asclaimed in claim 1, the image evaluation algorithm chosen is a machinelearning algorithm, the machine learning algorithm being a deep learningalgorithm, which is trained on a basis of overview contrast images withknown information.
 20. The method as claimed in claim 19, furthercomprising: using a deep learning algorithm based on a convolutionalneural network to identify a type of the sample carrier, performing asemantic segmentation for localizing structures of the sample carrier orof the sample, using a deep learning algorithm based on a fullyconvolutional network.